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ABSTRACT 

In March and April of 1992, 60 representatives of 
state-funded workplace education programs in New York attended 
day-long regional workshops to identify ways of improving program 
planning and evaluation methods. After the workshops, the 
participants returned to their home programs, organized local 
evaluation teams, and developed action plans to clarify their 
programs' goals, strengths, and problems and to suggest possible 
improvement strategies. In May 1992, the participants reconvened to 
present their draft action plans and the team process used to develop 
the plans. As a result of the workshop project, many of the action 
plans developed were incorporated into revised program funding 
proposals, several representatives who had participated in the 
workshops undertook special team^based planning and evaluation 
projects. New York's State Education Department began to revise its 
application format for workplace program funding, and many program 
participants developed links with other workplace educators in the 
United States and Canada. Possible future actions in the areas of 
staff development, collaborative site-level planning and evaluation, 
and collaborative statewide policy development were identified. 
(MN) 
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EXECUTIVE SUMMARY 

In 1992, the New York State Department of Education (SED) 
turned its attention to tiie question of "how can we improve how 
evaluation is carried out in state-funded workplace basic skills 
programs?" SED recognized tiiat, too often, "evaluation" consisted of a 
ritual in which program administrators collected "data" for the 
purpose of convincing SED that the programs warranted further 
funding. 

SED realized that evaluation could be a much more valuable 
tool if used by practitioners to strengthen program practice. With 
the state AFL-CIO, SED embarked on a project designed to get 
practitioners to re-rhink how they evaluated their programs and 
develop more effective evaluation procedures. 

hi March and April of 1992, some sixty representatives of 
state-funded workplace education programs convened for day-long 
workshops in their respective regions. In those meetings, 
participants first analyzed their current evaluation practices, 
identifying ' v^at worked" and \\hat needed improvement 
Participants tiien tried out a "team" evaluation activity, working in 
focus groups to analyze the purposes, strengths, limitations, and 
needed unprovements of SED's workplace Uteracy initiative. The 
day-long workshops ended with an assignment for the participants: 
They were to return home, organize an evaluation team in their 
programs, and develop an action plan. This plan was to clarify tiie 
program's goals, strengths, problems, and steps which the program 
would take to improve itself in the coming year. This action plan 
would ultimately be incorporated into the program's next funding 
proposal to the state. 

After those initial workshops, participants carried out the 
above assignment, then reconvened in May to report to then" fellow 
practitioners. Each program presented (1) the draft action plan it 
had developed in tiie previous month and (2) tiie team process each 
site went through to develop that plan. 

These early-1992 workshops influenced a niunber of 
subsequent efforts to develop new forms of evaluation: 

L 
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• When programs submitted their funding proposals to SED in 
June 1992, they incorporated revised versions of the above team- 
developed action plans. 

• From mid-1992 to mid-1993, several programs undertook 
special "team-based" planning and evaluation projects. Site-level 
teams were organized for the purposes of (1) clarifying the needs 
and interests of both organizations and individuals, (2) napping 
out various ways the organization might respond to these needs, 
and (3) generating understanding and buy-in by the full range of 
stakeholders represented in each site. 

• SED has begim to revise what it asks its funded workplace 
programs to include in their funding appUcations. SED is now 
asking for greater evidence of vAiax programs will do to 
"continuously improve" themselves. 

• Those involved in the above activities have developed hnks 
with other workplace educators in the U.S. and Canada wiio are 
also attempting to create similar forms of collaborative planning 
and evaluation. These people are cross-fertilizing each otiier via 
specially-organized workshops, meetings, telephone conversations, 
and sharing of resource materials. 

These activities constitute experiments in staff development, 
site-level planning and evaluation, and state-level policy 
development An analysis of each of these experiments suggests the 
following lessons and possible future actions: 

<;t?^ fFriPvelor>ment : The 1992 workshops focused on a topic of 
concern to many practitioners. The workshops' participatory 
nature allowed participants to build on what they know and - 
with support from each other ~ begin to develop more 
meaninghil evaluation and planning methodologies. More such 
staff development activities should be implemented, to develop a 
pool of experienced practitioners - and other stakeholders (e.g., 
Sii)loyers, unions) - skilled in effective planning and evaluation. 
In addition to workshops, informal communications, directories, 
newsletters, training videos, internships, consultancies, and 
practitioner research projects might be used to develop necessary 
expertise, 

(- Qllflhorative fjirHfVfl rl?^nn^"g evaluation: While the new 
emphasis on "collaborative" planning and evaluation holds great 

• » 

u. 



promise, it also requires time, e:q)ertise, and commitment This is 
true not only for the educators involved but for all other 
stakeholders as welL Those going into team-based planning must 
be aware of what is required to make a team work. They must 
also be given the time and resources to develop tiieir expertise by 
actually getting involved in the creation and mairitenance of a 
team process. 

Collaborative statewide polW development: The 1992 workshops 
and the subsequent action plans vMch programs submitted to SED 
were examples of programs giving helpful feedback to SED. This 
spirit of collaborative policy development should be further 
encouraged by SED in the forms of meetings, site visits, and 
funding applications. Practitioners should themselves take more 
proactive roles in influencing the future of state funding and 
guidelines for workplace education programs. 

Imbedded in these efforts is a new way of looking at workplace 
education. Rather than seeing education as a means of serving 
narrow, short-term, individual interests, leaders in New York's 
workplace education community are increasingly seeing it as a 
vehicle for longer-term, collective benefit for entire organizations. 
This perspective on workplace education corresponds directly to the 
principles of team-based, continuous-improvement management 

This new way of thinking - and the practitioners who are 
creating new practices to go with it ~ deserve the support of those 
with a real interest in improving how work is done in New York 
State. 
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Introduction 



In 1991, policy makers in the New York State workplace 
literacy initiative recognized tbat, for workplace educators, 
"evaluation" too often came down to preparing a report for funders 
vsiiose primary purpose was extolling the program's virtues. The 
initiative's policy makers recognized that evaluation could be much 
more - a dynamic tool for creating strong programs. But not 
enough attention was being paid to creating more-meaningful ways 
of using evaluation and getting those new evaluation tools into the 
hands of people at the site level who could benefit from them. 

Late that year, Robert Knower (Coordinator of Workplace 
Literacy for the New York State Education Department) and Dawn 
Krusemark (Coordinator of Workplace Learning Programs at the New 
York State AFL-CIO) met with workplace literacy specialist Paul 
Jurmo. Dr. Jurmo had, with evaluation specialist Laura Sperazi, 
recently concluded a special evaluation project for the 
Massachusetts Workplace Education Initiative. In that project, 
planning and evaluation teams were created in six workplace 
programs. These teams planned and implemented their own 
evaluations, thereby not only gathering information of importance to 
the stakeholders represented on the team but enhancing 
understanding and support for the program among those 
stakeholders. 

It was agreed that Dr. Jurmo ("The Consultant") would, in early 
1992, disseminate to New York State workplace educators some of 
the experience gained in the Massachusetts project. This report 
describes the workshops and site-level evaluations vMch resulted in 
the first she months of 1992, as well as longer-term follow-up 
activities taken by several participants in those workshops. It 
concludes by recommending actions vMch stakeholders might take 
to create more effective strategies for planning and evaluating New 
York's workplace education efforts. 
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Purposes of the project 



The Consultant was to help New York State Education 
Department (SED) staff organize and facilitate a series of workshops 
and related support services for key representatives of workplace 
basic skills programs funded by the Department These activities 
were to: 

1. Enable participants to create and facilitate "evaluation 
teams" in their respective programs. 

2. Enable participants to work with their respective teams to 
plan and implement data-gathering activities. In these 
activities, the teams would collect information which they 
would use to develop action plans for program improvements. 

3. Enable participants and tiieir respective teams to also 
develop action plans vMch could be provided to program 
funders (ir.cluding NYSED) and other interested outside parties. 

4. Develop a corps of practitioners experienced in effective 
evaluation methodologies, along with sample evaluation tools 
and other resource materials. These human and material 
resources could then be used by other workplace educators in 
the state and nation to improve their evaluation-related skills. 



What happened in early 1992 



L Initial workshops: March-April 1992 

In March and April 1992, more than sixty representatives of 
twenty NYSED-funded workplace literacy programs met in day-loug 
workshops in their respective regions. These workshops were labor- 
intensive for the participants, as they were asked to (1) analyze what 
they were already doing in Uie area of evaluation and how to 
improve it, (2) practice a collaborative evaluation activity through 
\>hich they could evaluate the state's workplace education initiative, 
and (3) prepare to cany out a pilot "team" evaluation of their own 
local-level programs. 

Analyzing evaluation practice in New York State to date 

In the morning of the workshops, the participants first 
developed a lengthy list of activities currently used in their 
programs which could be categorized as evaluation-related. These 
included standardized tests; informal learner interest inventories; 
informal feedback from various stakeholder groups; pre-, interim, 
and post-questionnaires for learners; observation by program 
administrators; computerized instructional program with built-in 
monitoring system; and review of classroom and workplace 
records. 

As they described these practices, the participants also 
described the purpose for each activit>'. These purposes can be 
simimarized as: 

( 1) to clarify learner needs and interests and program 
objectives, 

(2) to determine what impacts the program has in fact had, 

(3) to help various stakeholders decide whether to continue to 
support (or get involved in) the program in the future, 

(4) to help decide how to improve the program in the future, 

(5) to provide data required by the funder, 

(6) to get management and union involved in monitoring the 
program, and 

(7) to clarify what needs to be done to manage a culturally 
diverse workforce. 



iO 

3 



ERIC 



During the same morning, participants analyzed their current 
evaluation practices in terms of "strengths" ("^t works"), 
"limitations" ("\^«laf s not working," "obstacles," "problems"), and 
"vsMt needs to be improved." listed below are some of the key 
improvements \^hich participants felt need to be made in current 
evaluation practice: 

• Lengthen the funding cycle to allow time to produce 
documentable results. Longitudinal impact should also be 
measured, not just short-term impact 

• Clarify what information stakeholders want and then develop 
data-gathering instruments to foe \s on those particular types 
of information. 

• De-emphasize head counts and standardized test scores as 
indicators of program impact, as they are of litde relevance to 
management, union, and learner concerns. Standardized tests 
also tend to "turn off' participants and don't take into account 
test-taking anxiety and cultural diversity. 

• Recognize that some objectives (e.g., self-esteem, learning to 
learn, problem-solving) are difficult to measure. 

• Be careful not to assume a direct causal relationship between 
a basic skills program and worker job performance. 

• Set up evaluation teams composed of carefully-selected 
representatives of stakeholder groups. Members should have 
sufficient knowledge and authority to make informed decisions, 
as well as good communication links to their "consituendes." 
But be careful not to set up planning and evaluation structures 
which are too time-consuming for stakeholders. 

• Be careful that all stakeholders give input into and 
imderstand the purposes of an evaluation. 

• All stakeholders (including the state) should clarify program 
goals first before initiating any evaluation. Goals can include 
job-related development and/or personal development 

• Introduce the "quality" concept as a guiding principle for 
wliat work organizations, education programs - and education 
program evaluations - should be focusing on. 

• Beware of using evaluations to eliminate or otherwise hurt 
workers. Build trust and confidentiality into evaluation 
activities. . 

• Make sure that stakeholders actually use the mformation 
generated by an evaluation rather than letting it "sit on the 
shelf." 

II 



• Recognize that (1) good evaluation can be very valuable, (2) 
it takes time, and (3) those doing it should be paid for their 
time. 

• Recognize the role which ciltural diversity needs to play not 
only in work organizations but in workplace education 
programs and program evaluations. 

• Provide training and other forms of support for practitioners 
trying to do the above. 

Practicing a collaborative evalua tion activity to evaluate 
the state*s workplace education initiative 

In the afternoon of the initial workshop, participants were 
given an opportunity to practice doing a "team" evaluation 
activity-. They were asked to randomly divide into smaller groups 
and develop a group response to the following discussion 
questions: 

1. What are the goals/objectives of the state workplace 
education initiative? 

2. What are the strengths of the initiative to date? 

3. What are its limitations to date? 

4. What actions should be taken to improve the initiative, and 
who should take those actions? 

By going through this activity, the participants were to (1) get 
some practice using a focus group format to analyze a program in 
vMch they were all involved, and in the process (2) provide some 
useful feedback to policy makers and others involved in the 
state's workplace education initiative. 

The participants' responses to these discussion questions are 
siunmarized below: 

Question #1: What are the goals /objectives of the state 
workplace e ducation initiative? 

• To educate the workforce, improving job-performance 
through enhanced general basic skills and job-related skills. 

• To keep jobs in NY State. 

• To help NY companies make transition to new technology. 

• To encourage labor-management-education cooperation. 

• To compensate for what the formal education system 
doesn't do. 
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• To improve the welfare of workers, their families, and 
their communities. 

•To help employers recognize the role of basic skills and 
integrate basic education into an overall training strategy. 

• To encourage employer investment in worker education. 

• To reduce social co^ts of unemployment and dislocation. 

• To develop models of contextualized learning. 

OiiPstion #2; What are the fftrpngths of thp initiative tQ date? 

• The workforce gets an educational opportunity, thereby 
enhancing both the job-related and personal development of 
individual workers and the development of workers' 
families and communities. 

• Practitioners develop expertise and "cross-fertilize" each 
other. 

• Each program has autonomy to define its own needs. 

• Rexibility encourages innovative models. 

• Demonstrates models of labor-management-education 
collaboration. 

• Employers and employees increasingly recognize value ot 
education. 

• Productivity of participating workplaces has increased. 

• Has brought togethei' various cultural groups in a mutually 
constructive way. 

• Has helped introduce some workplaces to "quality" 

concept , , 

• Has established a funding source dedicated to workplace 

literacy education. 

• Provides a tool for union recruitment 

Qiipstion What are t hP initiative's limitations to date? 

• Communication within tiie initiative is not timely enough 
(e.g., upstate programs feel they .don't get the same 
information NY Qty programs receive.; 

• Guidelines fix)m NYSED are not suffidendy clearly defined 
and are developed without input from the field. Result is 
bad data and too much paper. 

• Programs not clear what information NYSED wants from 
them 

• Funding is late in reaching programs. 
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• It is difficult or misleading to compare programs within 
the initiative, as they are all different (focusing on different 
goals, etc.) 

• Outcomes related to job performance are difficult to gauge. 

• Costs (e.g., staff time spent on upfront negotiations) picked 
up by local providers can be considerable. 

• Not enough money is available for basic functions like new 
program start up, etc. 

• Not enough opportunity to cross-fertilize across programs 
(e.g., to share innovative ideas for instructional design). 

• Artificial requirement to use a partioilar instructional 
design methodology and to divide services according to the 
60-20-20 formula. 

• Company management has frequent turnover, making 
continuity and communication difficult 

• Educators not always sensitive to learner needs and 
interests. Learners (and the cultural groups they come 
from) not involved in the program development process. 

• Providers not always clear vAio (e.g., provider, union, or 
management) should be responsible for recruitment. 

• Not clear whether to give release time and, if so, how to 
require it of employers. 

• Hard to respond to multiple skill levels in one program. 

• Retention rates sometimes low. 

• A weak economy limits companies' willingness to provide 
release time, promotional opportunities, and other needed 
supports. 

• Hard to "sell" the idea of workplace education to labor and 
management 

• Duplication of effort and turf issues. 

• Social stigma inhibits worker participation. 

• Fear that management might misuse information from 
evaliiations/assessments. 

• Program not integrated into other adult education, job 
training, and social service programs. 

• Companies not picking up funding after the initial state 
seed money. 

Question #4! What actions should be taken to improve the 
initiative. ar >d whO those actions? 

• NYSED should develop clear, not-overly-cumbersome 
guidelines, with input from local providers, allowing greater 



flexibility in terms of timing and course content. Solid 
guidelines will show providers and other stakeholders how 
best to plan, implement, and evaluate their programs. 

• At local level, involve all stakeholders in program 
planning, to encourage their investment 

• Communicate requirements to the local level in a timely 

way. ' 

• Mechanisms for networking/communication among 
providers should be established (e.g., on regional basis, via 
clearinghouse, directory, case studies, etc.) 

• Use Regional Education Centers as a conduit for state 
funds. 

• Alternative methods of determining program success need 
to be developed, with input from field. Develop ways of 
mutual feedback among all stakeholders. 

• Providers need to have greater access to information and 
training, relying on expertise within state wiien possible. 
Topics should include innovative practices like "team" 
planning, etc. 

• Fimding should be continued and increased - not 
decreased. 

• Funding is needed not only for instructional hours but for 
planning, evaluation, staff selection and training, learner 
recruitment, etc. 

• Develop guidelines on v/hat constitutes a viable site to 
work with. 

• Help programs develop marketing strategies. 

• Institute a lobbying campaign with state, federal, imion, 
corporate. Chambers of Commerce, job-training agencies, and 
other funding sources. 

• Educate employers about tiie need for "creative solutions" 
and longer-term investment in solid education programs 
and transformation of the workplace. Develop tax incentives 
and other ways of encouraging employe! investment 

• Develop better communication and coordination among 
state agencies involved with worker education and training. 

• Develop positive "public relations" (and training for all 
stakeholder groups) aroimd the worker education issue, to 
reduce stigma and encourage paTucipation by all 
stakeholders. 

• Stirvey employers to clarify what skills they now need in 
the emerging workplace. 

8 '5 



• Employers and unions might pay a stipend to cover some 
of the costs of workplace education services funded by the 
state. 

• Worker education should be integrated into a model of 
ongoing development of jobs and technology, rather than be 
seen as a one-time, isolated phenomenon. 

Preparing to carry out a pilot "tpam" Pvalnatinn of their 
own local-lAv**1 prnprflms 

Prior to departing from the initial workshop, participants were 
introduced to a "team" evaluation model outlined in a handbook 
specially designed by the Consultant This model was based on 
work done the previous year in Massachusetts. In it, the 
stakeholders at a workplace basic skills program site would - as a 
team ~ plan and carry out an evaluation wliLch would focus on 
information of particular relevance to members of that team. 

The Consxiltant gave a brief overview of the steps an evaluation 
team woxild go through. He then presented the workshop 
participants with an "assignment" to cany out over the following 
4-6 weeks. Once back at their sites, the participants were to 
convene a focus group composed of representatives of the 
program's stakeholder groups. In this group, stakeholders were 
to develop statements on: 

1. what the stakeholders expected the program to achieve; 

2. to what 4*xtent those expectations had been met to date; 

3. the progi-am's "strengths" (factors contributing to program 
success) and "limitations" (obstacles or problems encoimtered); 

4. actions vMch needed to be taken to improve the program 
during the coming funding cycle. 

This assignment was designed to (1) provide participants with 
an opportuity to carry out a"team" evaluation activity and (2) 
provide SED with action plans from each site to consider vAien 
deciding on funding to be provided to those sites in the coming 
year. 
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n. T ry^"g the evaluat lfin tPatn model! April - May 
1992 



During the foUowing 3-6 weeks, participants puUed together 
the team evaluation activity outlined in the assignment above They 
not only developed the action plans asked for in the assignment, but 
documented the process they went through to develop those action 
plans. Although they were given the option of getting additional 
feedback and guidance from the Consultant, SED, and state AFL-CIO 
by telephone, few participants actually requested such assistance. 

m. Fnllow-up workshops : May 1992 

In May, the participants reassembled for a follow-up workshop 
in their respective regions. There, each reported on two things: (1) 
their action plans (i.e., the "products" they developed in the previous 
month) and (2) the "process" they went through at each site. 

The content and quality of both the products and processes 
varied from site to site. These variations depended on a number of 
factors: 

• TTi ^ vn llin gness and avaj lahilitv of orTipr stakpholders to 
participate : Some sites had a history of working collaboratively 
with partners (e.g., managers, union representatives) to monitor 
program progress. In such cases, those partners understood the 
value of holding a focus group to outline a plan of action for the 
coming year, hi some other sites, partners had not been asked to 
fulfill such roles before, and resisted when asked to do so, due to 
shyness, suspcion, or simple lack of understanding of the potential 
benefits. Also, in some cases partners were not available to 
participate due to scheduling conflicts. (Only a relatively short 
time was given to participants to organize this focus group 
activity.) 

• -Qif pv pprience and persoec Hve of the partirinants themselves: 
In terms of their experience, the participants could be divided 
into three rough categories: (1) those with Uttle experience in 
workplace literacy and/or with a preference for traditional 
approaches to evaluation, (2) those with a willingnew to try 
something new in workplace education evaluation, even if they 
had only limited experience in the field, and (3) those with 

17 
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relatively more experience in workplace education and sl 
vsdllingness to try something new in the area of evaluation. Those 
v>^o made the strongest presentations at the follow-up workshop 
understandably tended to come from the last of these three 
groups. 

After the participants described their respective action plans 
and the processes they went through, the Consultant and die 
representatives from SED and .\FL-CIO had the opportunity to 
give feedback to each participant. Participants were also asked to 
give feedback to each other, as welL 

It was agreed that the participants would now have 2-3 weeks 
to return home and revise their action plans. These revised plans 
would be submitted to SED in Jime as part of the participants' 
flnn^iai funding proposal. 
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Subsequent actions 
taken by individual programs 



Arrinn plans suhmitted Tft ^^^^^ 

In June 1992, the participants submitted revised versions of 
their action plans to SED as part of their annual funding proposals. 
The content of the action plans varied from generally-unfocused 
collections of facts and statistics to plans much more directiy focusing 
on the information asked for in tiie assignment 

In one example of a weaker "plan," the provider provided a 
rambling history of the program, with multiple attachmejits 
(including a course syllabus and a sample post-course questionnaire 
for students) but never analyzed the program's strengths, limitations, 
and needed improvements, as requested. In an example of a 
stronger plan, there was a much clearer focus on the requested 
information. (It y/sls not clear, however, how that information was 
actually developed. One could speculate that the plan might have 
primarily been the work of the education provider with only limited 
input from other members of the "team" described on page 1.) 

Several participants used computer-generated grids to lay out 
various tasks to be accomplished in the coming year. 

Another observation: The language used in many of the action 
plans was not a good example of "clear language." Sentences tended 
to be long, full of jargon, and - intentionally or not ~ vague. One 
example of vagueness: indicating that "assessment" will be done 
without indicating more specifically how data will be gathered and 
what type of data will be generated. 

A reader of the weaker action plans could conclude that ( 1) the 
writers need a course in "clear writing," (2) the writers don't fully 
understand vs^t it takes to plan a solid program, and they simply 
use jargon to disguise that fact, or (3) the writers have a good grasp 
of VN^iat a solid program requires but aren't very concerned how 
clear such a plan is as long as they get funded. 
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"Tfarr planning arid gvaluatjon* j^^^aft irioH nut In a fnimher 
of sites 



In the year since the 1992 workshops, the concept of team 
planning and evaluation has been tried out in a number of New York 
State workplace education projects. The 1992 workshops to varying 
degrees helped shape the thinking of the practitioners organizing 
those programs. Described below are the experiences of several of 
these attempts iDO approach planning and evaluation from a 
collaborative perspective: 

RDTI?; of Dutrhess County 

Shirley Toth, workplace literacy coordinator at BOCES of 
Dutchess County, has for some time operated on the premise that, 
"to give the customers v/hat they want, you have to do a lot of up- 
front planning. . . The heart of the matter is in the planning, not 
just the doing." 

For her program for county workers, she organized a "training 
committee" composed of heads of each participating department 
This committee's role was to set program goals and to then decide 
"how do we know this program makes a difference?" In other 
words, the committee was to establish criteria for assessing 
program impact 

The coordinator admits that getting such a committee to be 
clear about their expectations for the program is easier said than 
done. Too often, a workplace literacy program is seen as a "perk" 
(a "feel-good course") for workers rather than something that is 
supposed to produce clear, documentable results. It might be 
argued that such an approach allows learners and teachers to set 
their own goals without having to worry about satisfying anyone 
else's expectations. However, it might also be argued that, if the 
program cannot doomient clear impaa on, say, learner job 
performance, then it will be hard to justify further funding vihexi 
the next education budget comes up for review. 

The coordinator says that stakeholders need to be educated 
about the value of sound evaluation. Such evaluation would 
require stakeholders to be clear about ^^^t they hope the 
program will achieve. A soimd evaluation would then gather 
information which would demonstrate to what extent those goals 
in fact were being achieved. And, to ensure understanding of the 
program by the stakeholders, all stakeholders might be invol . ed 
in the process of gathering information, as well. 
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The coordinator feds tbat such an approach to planning and 
evaluation vworks best in a workplace conunitted to the principles 
of quality and employee involvement 

<^T|<IY at Stony Brook 

In early 1993, REACH (a collaboration of the Governor's Office 
of Employee Relations [GOER] and the Civil Service Employees 
Association [CSEA]) hired workplace education consultant Munel 
Medina to conduct an organizational needs analysis (ONA) for 
residence hall employees at the State University of New York in 
Stony Brook. The ONA methodology used was drawn from work 
done by Canadian workplace education specialist. Sue Folinsbee. 

It takes a broad look at the role of basic skills in the 
organization. It clarifies not only \Ntoether employees are in need 
of basic skills upgrading but also whether the organization itself 
needs to take steps other than basic skills education to make best 
use of its workforce. The ONA methodology assumes that, by 
involving key stakeholders in analyzing v^ere basic skills 
education and other initiatives mi^it (or might not) be needed, a 
more realistic perspective on basic skills education will be 
developed. Such a collaborative approach also aims at buUding in 
stakeholders' support from the very beginning of the program. 

At Stony Brook, Dr. Medina worked with an existing advisory 
committee composed of representatives of CSEA, GOER, and the 
University's Division of Residence Operations. She conducted focus 
groups and individual interviews with employees, supervisors, 
and advisory committee members. Those discussions indicated 
that the organization should not institute a "basic literacy 
program" in the traditional, stand-alone sense. Rather, workers 
might participate in a range of job-related reading, writing, math, 
ESL, and technical training courses. Other possible initiatives 
identified; publishing of a book of maintenance terms, a workshop 
for students living in the halls to familiarize them with the 
employees who care for the facilities, an oral communication 
course, basic computer training, a stress management course, and 
job promotion opportunities. 

Dr. Medina feels that the process not only expanded 
stakeholders' thinking on yAiat "basic skills" might mean within 
the organization, but also got people - including workers, 
supervisors, and higher-level managers - involved in the 
planning process. This process also lays a foundation for 
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vs^tever curriculum design work is to follow, as it identifies 
much more clearly \s4iat stakeholders' needs and interests. 

The Division's management is currendy considering the report's 
recommendations. 

STINYar Albany 

As at SUNY Stony Brook, CSEA and GOER hired a workplace 
education consultant, Carol Young, to conduct an organizational 
needs analysis for the State University of New York at Albany. 
She worked with a committee composed of state-level 
representatives from GOER and CSEA, frontline workers from day 
and night shifts, and a supervisory janitor. 

To get the ONA process going. Dr. Young met with the advisory 
committee to explain the ONA process and with four work groups 
in their workplaces to make 10-minute presentations to 200 
workers. These latter work-group meetings enabled her to get the 
word out about why she would be conducting interviews and 
focus groups with workers. Those meetings also gave employees 
an opportimity to get to know her and to approach her informally 
to tell her vihax ba.sic skUls help they wanted. Dr. Young notes 
that this process "generated a lot of excitement, especially as you 
stick with it." The ONA process also "begins the intervention by 
building trust" 

Her analysis focused on the basic-skills-related needs of 
employees in the Department of Physical Plant Building and 
Institutional Services. Following essentially the same ONA format 
used at SUNY Stony Brook, the analysis identified custodial staffs 
basic skills needs and interests, possible educational services, 
other possible initiatives, and barriers and supports for learning. 

This analysis was I ased on information gathered in focus 
groups and interviews with custodial staff and their supervisors. 
Results were presented to the advisory committee, which 
strengthened it with additional suggestions of their own. The 
committee has now presented the report to senior management 
for their final approval. 

state Educati on Denartment 

SED has been revising vitax it wants its funded programs to 
include in their funding applications. Rather than focus so heavily 
on providing evidence of "return on investment," programs should 
now describe how state funds will be used to "continuously 
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improve" themselves. That is, vstoile SED will encourage programs 
to provide evidence of program impact, they should also go 
through a process - preferably a collaborative one - which will 
clarify vAiax steps will be taken to strengthen particular 
components of the program. In effect, then, SED is incorporating 
into its own operations tiie notion of "continuous improvement" 
vAdch it hopes local workplaces will use as a guiding principle. 

TinVs tnaHi * to Other prngrams iigtng ximiTar nianning and 
evaluation methodologies 

In addition to taking the initiative to organize the above 
projects, several workplace educators with an interest in alternative 
approaches to planning and evaluation have made links with 
resource groups outside the state, as follows: 

Informal northeast region working group : hi February 1993, 
representatives of REACH, ABC CANADA, and the Massachusetts 
Workplace Education Initiative met in Newton, Massachusetts in a 
one-day meeting organized by Paul Jurmo and his Massachusetts- 
based colleague, Laura SperazL (Dr. Jurmo and Ms. Sperazi are 
co-directors of a one-year project funded by the National Institute 
for literacy in which they are further refining the team-based 
evaluation process vMch Dr. Jurmo presented in the 1992 
workshops.) The participants all shared an interest in 
collaborative planning, evaluation, and assessment In the day- 
long meeting, participants shared information about their 
respective projects and agreed to possibly cooperate further via 
consultations, an electronic-mail bulletin board, and additional 
meetings. 

ABC CANADA: Having met Sue Folinsbee, workplace literacy 
specialist at ABC CANADA at the above meeting hi Newton, 
Massachusetts, REACH representatives arranged to have her 
conduct a one-day workshop in March 1993. The workshop 
introduced participants to the organizational needs analysis 
methodology which has since been used at SUNY Stony Brook and 
SUNY Albany. Participants were drawn primarily from the 
agencies involved in workplace literacy progtams sponsored by 
REACH. 




National Institute for literacy: CSEA is currently developing an 
alternative assessment methodology for workplace literacy 
programs under a grant from the National Institute for Literacy. 
The methodology uses basic ethnographic observation and 
interview techniques to clarify how potential learners actually use 
communication skills in their work. The resulting information can 
be used by curriculum developers to create curricula which focus 
more directly on the literacy functions which learners actually 
need and are interested in. This assessment project ties in with 
CSEA's larger interests in collaborative planning and evaluation. 
REACH and several other participants in the February meeting in 
Newton, Massachusetts have been exploring ways of cross- 
fertilizing across their respective projects. 

U.S. Department of Education: A nimiber of the programs 
described here - in New York and Massachusetts- vMch have 
taken a lead in creating the innovative approaches to worker 
education described here have been successful in getting 
additional funding from the National Workplace Literacy Program 
of the U.S. Department of Education. As such, they will serv e as 
demonstration projects through v\iiich tiliese innovations can be 
tried, analyzed, and disseminated nationally. 
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Lessons learned 
and 

possible future actions 



The above-described activities carried out in 1992 and 1993 
constitute experiments in staff development, site-level planning and 
evaluation, and state-level policy development Usted below for each 
of those areas are lessons learned and possible follow-up actions. 
This analysis draws heavily on ideas provided by practitioners vAio 
have taken a lead in developing collaborative approaches to 
planning, evaluation, and assessment 

While the author acknov\4edges that all of ti?e possible actions 
require money and staff time, money and staff are not the sole 
variables here. Also needed are a recognition that these efforts are 
valuable and a commitment to finding the funds and time to make 
them happen. 

As a model of staff development 

T^swns learned: A number of participants in the 1992 
workshops claimed that they were a refreshing, useful vehicle for 
staff development Reasons dted: 

• They focused on issues of concern to practitioners. 

• They encouraged participants to draw directly on their prior 
experience and to participate actively in developing strategies. 

• They allowed participants to try out new ideas in real 
contexts and then to reconvene to analyze tho^e efforts. 

• They fostered communication and a tescn idet tity among 
fellow practitioners within the state. 

But those participants also noted that coordinating effective 
planning at the site level requires that practitioners have special 
expertise, communication skills, commitment, flexibility, 
creativity, and patience. A wide number of practitioners need 
such qualities if there is to be continuity of program quality if 
there is turnover of education staff. 
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l^nssible actions : SED, REACH, and others interested in 
developing more effective approaches to planning, evaluation, and 
assessment should organize various activities aimed at developing 
a pool of workplace educators able to implement such ideas. 
Possible activities include: 

• Ongoing informal co i nnmniratiQns among PractiTioneTS Witb 
an intprpst in ff^Uflh^rati^** planning: This can be done via tiie 
E-mail system begun by die "Newton group," periodic 
infonnation-sharing meetings, an annotated directory of 
resource people (within and outside NY), a specially-designed 
newsletter, and/or dissemination of articles and other resource 
materials. 

• More-formalized trainin g oppornmities: A statewide or, 
possibly, regional workplace education conference might be 
organized focusing directiy on the need ~ and methodologies - 
for new ways of planning and evaluating programs. Rather 
than try to cover every possible issue in workplace literacy, the 
agenda, facilitators, and participants would be carefully 
selected to help participants create new, more effective 
strategies. Tr ainin g opportunities should be given not only to 
practitioners but other stakeholder groups, as well. Concise 1- 
day workshops, manuals, and/or videos might be developed for 
managers, unions, and workers vAdch would help thase 
stakeholders understand the purposes and roles involved in an 
education planning team. 

• Internships: Practitioners might spend time in each others' 
programs to learn from ~ and help shape - p l a nnin g and 
evaluation practices used in those programs. 

• Consultancies: Resource persons from within and outside the 
state might be made available to individual programs - or 
regional consortia of programs - to help them analyze and 
solve planning and evaluation-related problems. Such on-site 
visits by committed outsideresource persons can not only 
provide valuable technical assistance but boost the morale of 
local stakeholders vAiO might otherwise feel isolated. 

• Prarritioner research : Practitioners might take on the role of 
"practitioner as researcher" to document, analyze, and 
disseminate their planning and evaluation practices. 
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Information might take the form of case studies or coUections 
of sample data-gathering instruments. This information imgnt 
be disseminated via articles, conference presentations, or video 
presentations. This research might be assisted by luuversity- 
based research specialists, although emphasis should be on 
disseminating useful experience ratiier tiian hording of 
research findings in ivory towers. 

As a model of collaborative site-level planning and 
evaluation 

T ^ c^Ti^ ipamed : Practitioners vAiO have committed themselves 
to trying out collaborative planning and evaluation activities m 
their programs give the following feedback: 

• Even those programs which have tried to develop 
collaborative forms of planning, evaluation, and assessment 
recognize tiiat tiieir expertise is limited. It is safe to say that 
no one in tiie state has had experience carrying out a complete 
team-based planning process from tiie initial ONA s^ge 
tiirough program implementation and evaluation. The p lannin g 
cycle model presented at tiie March workshop in Albany (or 
some variation of it) should be further refined, to show tiie 
links between all phases of organizing a worker education 
program, (One practitioner says we should be aiming for a 

"seamless process.") ^ 

• Getting all stakeholders involved in settmg goals and 
monitoring progress is vital if programs are to actually respond 
to their interests. This includes higher-level managers, who 
need to be involved even if tiiey are "busy." (Such higher-level 
decision-makers have decision-making autiiority vsiiich is 
needed to enable tiie process to work and to ensure that 
meaningful follovs^up action is taken when tiie evaluation's 
findings are in.) Anotiier group which needs to be involved: 
workers vsiiose language skills or work schedules might make 
it difficult to participate in team meetings. Special efforts must 
be made to deal witii tiiese obstacles of time, language, culture, 

and so forth. , ^ , . ^ t 

• While the full range of stakeholders should have meanmgful 
roles in the planning process, care must also be taken not to 
involve too many stakeholders at once, as an overly-large team 
can lead to confusion and, simply, too much work. 
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• Fostering that involvement is easier said than done. It takes 
time, expertise, and commitment of all involved, especially the 
key coordinators at the site level Coordinators need to be 
careful not to dominate the process, vMle persiatentiy nudging 
it along. The team coordinator can do so by continually asking 
questions, helping team members understand the purposes of 
working together, keeping the team moving on task. 
Coordinators more experienced in formal education settings 
will likely have to develop new skills and a new mindset to 
take on this role. 

• Not all sites are equally prepared to take on the added work 
of forming and implementing an effective planning team. 
Those organizations with an active commitment to ''TQM"-type 
principles (e.g., team management for continuous 
improvement) are more likely to buy into the notion of 
collaborative planning for worker education. But even where a 
TQM philosophy is being implemented in the larger 
organization, stakeholders don't automatically see how such a 
philosophy might be applied to the planning of an employee 
education program. In such a case, extra care must be taken to 
help stakeholders see the link between "team management" 
and "team education." 

• While stakeholders in some sites might philosophically see a 
value in a collaborative approach to employee education, they 
might not be able to commit themselves to a solid program 
because the organization itself is undergoing staff and budget 
cutbacks and other changes disruptive to an education 
program. 

• While there are many benefits to a "quality team" approach 
to education, it can also sometimes uncover previously- 
suppressed problems within the organization. Stakeholders 
for this reason somedmes resist what on the surface seems like 
a reasonable, good idea. 

• The very process of getting a team of stakeholders to 
critically think about what the organization needs to do to 
improve itself is itself a valuable learning experience. This is 
true VNhether or not the process leads to the creation of a 
worker basic skills program. 

• Instruaors should be involved in all phases of the 
collaborative planning process, to enable them to fully 
understand the context and people they are to be working in. 
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• By involving a team in analyzing the information gathered in 
an ONA and/or evaluation, the biases of any individual team 
member can be better controlled. 

pnxRiMp actions: 

• All stakeholders - especially funders - need to recognize 
the value of good, collaborative planning and the extra time 
such planning requires. Funds need to be set aside (especially 
for practitioners) to support such work. (Otherwise, as one 
practitioner put it, "the grant asks you to do the right job, but it 
doesn't let you prepare to do it") Funders should actually 
require that stakeholders commit time to upfront and ongoing 
collaborative planning. (The Massachusetts state guidelines 
might be consulted as a model of how a state can foster active 
involvement of all stakeholders in program planning.) 

• Guidelines should also be developed for practitioners to help 
them clarify (1) vstiether a site is ready to commit itself to a 
good program and, if so, (2) all the steps that must be attended 
to to plan and monitor an effective program. 

• Resources need to be set aside to create the above-described 
staff development opportunities to enable practitioners to do 

this work. . , 

• These forms of collaborative planning should be recognized 
as an important vehicle for labor-management cooperation. 
Rather than take an adversarial position, unions and employers 
have an opportunity to negotiate goals and build an effort 
vsfeich will move both the organization and employees ahead. 
Unions should also see this form of education as an opportunity 
to provide a valuable benefit to members, a benefit especially 
valuable in an era of wage freezes, staff cutbacks, and 
flattening organizational structures. 

• Special efforts must be made to clarify reasonable, 
meaningful roles for all stakeholders - including learners - in 
the planning process. The process needs to be broken down 
into easy-to-understand and easy-to-do steps which team 
members can feel comfortable with. 
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As a model of roUabor alive statewide Dolicv 
development 



Lessons learned : One feature of the 1992 workshops was a 
"feedback session" in vMdi practitioners suggested possible steps 
for improving the state's workplace education efforts. SED has 
incorporated some of those suggestions into its policies and 
practices. 

PofiRihlp actions: 

State Educarion Department : SED might review the statements 
which emerged from the 1992 workshops as well as the 
recommendations in this report SED might continue to move in 
the direction of "collaborative continuous improvement" and 
develop ongoing mechanisms for involving stakeholders in the 
shaping of effective workplace education policy. 

For example, the staff development workshops described 
above could have, as part of their agendas, mutual feedback 
sessions in \N^ch SED listens to practitioners' suggestions about 
how to improve programs - and vice versa. 

Funding proposals might include sections in v\iiich 
practitioners not only describe steps they will be taking to 
strengthen their programs internally but provide suggestions 
to SED about how it and other institutions might better support 
the field. 

SED should foster commimication not only with workplace 
education practitioners but with other stakeholders — 
employers, unions, and workers themselves ~ as well. 

Such commimications with the full range of stakeholders 
can help create guidelines and funding to ensure the 
continuous improvement of the field. 

SED might also take the principle of collaboration beyond 
the state's borders to draw on the experience of other leading 
states like nearby Massachusetts. 

SED should work with practitioners to develop a 
consistent, clear format wiiich practitioners can use to apply for 
funds and report progress. 
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Workplace ednrarimi practitioners: Practitioners need to be 
more proactive in helping to shape policy and practice not only 
at the level of their own programs but at the state level, as 
well. I'l a time wfeen state funds are limited, practitioners 
cannot lo3k solely to SED as a source of support They need to 
look to each other, to other practitioners aroxmd the country 
(and hx Canada), to their legislators, and to other public and 
private sector sources for material, technical, and moral 
support 
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Conclusion 



The above-described efforts in 1992 and 1993 reflect a 
growing willingness and ability witbin the state's workplace^ 
education community to rethink "planning" and "evaluation." 
Practitioners are beginning to collectively analyze what needs to be 
done to improve the field - and then act on that analysis to create 
new opportunities. 

These efforts are characterized by a new way of looking at 
workplace education. Rather than seeing education as a means of 
sendng narrow, short-term, individual interests, leaders in the field 
are increasingly seeing it as a vehicle for longer-term, collective 
benefit for entire organizations. 

Those with a real interest in improving how work is done in 
New York should support this new way of viewing employee 
education. They should also support those demonstrating the 
commitment and creativity to make that vision real. 
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